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Abstract 

During  the  support  period  July  1,  2011  -  June  30,  2012,  seven  research  papers  were  published. 

They  consist  of  three  types: 

•  Research  that  directly  addresses  the  kernel  selection  problem  in  machine  learning  [1,  2]. 

•  Research  that  closely  relates  to  the  fundamental  issues  of  the  proposed  research  of  this 
grant  [3,  4,  5,  6]. 

•  Research  that  is  in  the  general  context  of  computational  mathematics  [7]. 

Paper  [1]  studies  the  construction  of  a  refinement  kernel  for  a  given  operator- valued  reproducing 
kernel  such  that  the  vector- valued  reproducing  kernel  Hilbert  space  of  the  refinement  kernel  contains 
that  of  the  given  kernel  as  a  subspace.  The  study  is  motivated  from  the  need  of  updating  the 
current  operator-valued  reproducing  kernel  in  multi-task  learning  when  underfitting  or  overfitting 
occurs.  Numerical  simulations  confirm  that  the  established  refinement  kernel  method  is  able  to 
meet  this  need.  Various  characterizations  are  provided  based  on  feature  maps  and  vector-valued 
integral  representations  of  operator-valued  reproducing  kernels.  Concrete  examples  of  refining 
translation  invariant  and  finite  Hilbert-Schmidt  operator-valued  reproducing  kernels  are  provided. 
Other  examples  include  refinement  of  Hessian  of  scalar-valued  translation-invariant  kernels  and 
transformation  kernels.  Existence  and  properties  of  operator-valued  reproducing  kernels  preserved 
during  the  refinement  process  are  also  investigated. 

Motivated  by  the  importance  of  kernel-based  methods  for  multi-task  learning,  we  provide  in 
[2]  a  complete  characterization  of  multi-task  finite  rank  kernels  in  terms  of  the  positivity  of  what 
we  call  its  associated  characteristic  operator.  Consequently,  we  are  led  to  establishing  that  every 
continuous  multitask  kernel,  defined  on  a  cube  in  an  Euclidean  space,  not  only  can  be  uniformly 
approximated  by  multi-task  polynomial  kernels,  but  also  can  be  extended  as  a  multi-task  kernel  to 
all  of  the  Euclidean  space.  Finally,  we  discuss  the  interpolation  of  multi-task  kernels  by  multi-task 
finite  rank  kernels. 

Multiscale  collocation  methods  are  developed  in  [3]  for  solving  a  system  of  integral  equations 
which  is  a  reformulation  of  the  Tikhonov-regularized  second-kind  equation  of  an  ill-posed  integral 
equation  of  the  first  kind.  This  problem  is  closely  related  to  regularization  problems  in  machine 
learning.  Direct  numerical  solutions  of  the  Tikhonov  regularization  equation  require  one  to  gener¬ 
ate  a  matrix  representation  of  the  composition  of  the  conjugate  operator  with  its  original  integral 
operator.  Generating  such  a  matrix  is  computationally  costly.  To  overcome  this  challenging  com¬ 
putational  issue,  rather  than  directly  solving  the  Tikhonov-regularized  equation,  we  propose  to 
solve  an  equivalent  coupled  system  of  integral  equations.  We  apply  a  multiscale  collocation  method 
with  a  matrix  compression  strategy  to  discretize  the  system  of  integral  equations  and  then  use  the 
multilevel  augmentation  method  to  solve  the  resulting  discrete  system.  A  priori  and  a  posteriori 
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parameter  choice  strategies  are  developed  for  thesemethods.  The  convergence  order  of  the  result¬ 
ing  regularized  solutions  is  estimated.  Numerical  experiments  are  presented  to  demonstrate  the 
approximation  accuracy  and  computational  efficiency  of  the  proposed  methods. 

Although  the  computational  method  developed  in  the  three  papers  [4,  5,  6]  are  for  image 
processing,  the  ideas  used  in  this  development  are  suitable  for  solving  regularization  problems  in 
machine  learning. 

The  paper  [4]  introduces  a  proximity  operator  framework  for  studying  the  Ll/TV  image  de¬ 
noting  model  which  minimizes  the  sum  of  a  data  fidelity  term  measured  in  the  Ll-norm  and  the 
total-variation  regularization  term.  Both  terms  in  the  model  are  non-differentiable.  This  causes 
algorithmic  difficulties  for  its  numerical  treatment.  To  overcome  the  difficulties,  we  formulate  the 
total- variation  as  a  composition  of  a  convex  function  (the  Ll-norm  or  the  L2-  norm)  and  the  first 
order  difference  operator,  and  then  express  the  solution  of  the  model  in  terms  of  the  proximity 
operator  of  the  composition.  By  developing  a  chain  rule  for  the  proximity  operator  of  the  com¬ 
position,  we  identify  the  solution  as  fixed  point  of  a  nonlinear  mapping  expressed  in  terms  of  the 
proximity  operator  of  the  Ll-norm  or  the  L2-norm,  each  of  which  is  explicitly  given.  This  for¬ 
mulation  naturally  leads  to  fixed-point  algorithms  for  the  numerical  treatment  of  the  model.  We 
propose  an  alternative  model  by  replacing  the  non-differentiable  convex  function  in  the  formulation 
of  the  total  variation  with  its  differentiable  Moreau  envelope  and  develop  corresponding  fixed-point 
algorithms  for  solving  the  new  model.  When  partial  information  of  the  underlying  image  is  avail¬ 
able,  we  modify  the  model  by  adding  an  indicator  function  to  the  minimization  functional  and 
derive  its  corresponding  fixed-point  algorithms.  Numerical  experiments  are  conducted  to  test  the 
approximation  accuracy  and  computational  efficiency  of  the  proposed  algorithms.  Also,  we  provide 
a  comparison  of  our  approach  to  two  state-of-the-art  algorithms  available  in  the  literature.  Numer¬ 
ical  results  confirm  that  our  algorithms  perform  favorably,  in  terms  of  PSNR- values  and  CPU-time, 
in  comparison  to  the  two  algorithms. 

The  goal  of  the  paper  [5]  is  to  improve  the  computational  performance  of  the  proximity  algo¬ 
rithms  for  the  Ll/TV  denoising  model.  This  leads  us  to  a  new  characterization  of  all  solutions  to 
the  Ll/TV  model  via  fixed-point  equations  expressed  in  terms  of  the  proximity  operators.  Based 
upon  this  observation  we  develop  an  algorithm  for  solving  the  model  and  establish  its  convergence. 
Furthermore,  we  demonstrate  that  the  proposed  algorithm  can  be  accelerated  through  the  use  of 
the  componentwise  GaussSeidel  iteration  so  that  the  CPU  time  consumed  is  significantly  reduced. 
Numerical  experiments  using  the  proposed  algorithm  for  impulsive  noise  removal  are  included,  with 
a  comparison  to  three  recently  developed  algorithms.  The  numerical  results  show  that  while  the 
proposed  algorithm  enjoys  a  high  quality  of  the  restored  images,  as  the  other  three  known  algo¬ 
rithms  do,  it  performs  significantly  better  in  terms  of  computational  efficiency  measured  in  the 
CPU  time  consumed. 

We  propose  in  [6]  a  preconditioned  alternating  projection  algorithm  (PAPA)  for  solving  the  max¬ 
imum  a  posteriori  (MAP)  emission  computed  tomography  (ECT)  reconstruction  problem.  Specifi¬ 
cally,  we  formulate  the  reconstruction  problem  as  a  constrained  convex  optimization  problem  with 
the  total  variation  (TV)  regularization.  We  then  characterize  the  solution  of  the  constrained  convex 
optimization  problem  and  show  that  it  satisfies  a  system  of  fixedpoint  equations  defined  in  terms 
of  two  proximity  operators  raised  from  the  convex  functions  that  define  the  TV-norm  and  the  con¬ 
strain  involved  in  the  problem.  The  characterization  (of  the  solution)  via  the  proximity  operators 
that  define  two  projection  operators  naturally  leads  to  an  alternating  projection  algorithm  for  find¬ 
ing  the  solution.  For  efficient  numerical  computation,  we  introduce  to  the  alternating  projection 
algorithm  a  preconditioning  matrix  (the  EM-preconditioner)  for  the  dense  system  matrix  involved 
in  the  optimization  problem.  We  prove  theoretically  convergence  of  the  PAPA.  In  numerical  exper¬ 
iments,  performance  of  our  algorithms,  with  an  appropriately  selected  preconditioning  matrix,  is 
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compared  with  performance  of  the  conventional  MAP  expectation-maximization  (MAP-EM)  algo¬ 
rithm  with  TV  regularizer  (EM-TV)  and  that  of  the  recently  developed  nested  EM-TV  algorithm 
for  ECT  reconstruction.  Based  on  the  numerical  experiments  performed  in  this  work,  we  observe 
that  the  alternating  projection  algorithm  with  the  EM-preconditioner  outperforms  significantly  the 
EM-TV  in  all  aspects  including  the  convergence  speed,  the  noise  in  the  reconstructed  images  and 
the  image  quality.  It  also  outperforms  the  nested  EM-TV  in  the  convergence  speed  while  providing 
comparable  image  quality. 

We  consider  [7]  the  precondition  of  linear  systems  which  resulted  from  the  finite  volume  method 
(FVM)  for  elliptic  boundary  value  problems.  With  the  help  of  the  interpolation  operator  from  the 
trial  space  to  the  test  space  of  the  FVM  and  the  operator  induced  by  the  FVM  bilinear  form, 
we  show  that  both  wavelet  preconditioners  and  multilevel  preconditioners  designed  originally  for 
the  finite  element  method  (FEM)  of  a  boundary  value  problem  can  be  used  to  precondition  the 
FVM  of  the  same  boundary  value  problem.  We  prove  that  such  preconditioners  ensure  that  the 
resulting  coefficient  matrix  of  the  FVM  has  a  uniformly  bounded  condition  number.  We  present 
seven  numerical  examples  to  confirm  our  theoretical  findings. 
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